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Abstract 

In order to meet the anticipated future demand 
for air travel, the National Aeronautics and Space 
Administration (NASA) is investigating a new 
concept of operations known as Distributed Air- 
Ground Traffic Management (DAG-TM). Under the 
En Route Free Maneuvering component of DAG- 
TM, appropriately equipped “autonomous” aircraft 
self separate from other autonomous aircraft and from 
“managed” aircraft that continue to fly under today’s 
Instrument Flight Rules (IFR). Controllers provide 
separation services between IFR aircraft and assign 
traffic flow management constraints to all aircraft. 

To address concept feasibility issues pertaining 
to integrated air/ground operations at various traffic 
levels, NASA Ames and Langley Research Centers 
conducted a joint human-in-the-loop experiment. 
Professional airline pilots and air traffic controllers 
flew a total of 16 scenarios under four conditions: 
mixed autonomous/managed operations at three 
traffic levels and a baseline all-managed condition at 
the lowest traffic level. These scenarios included en 
route flights and descents to a terminal area meter fix 
in airspace modeled after the Dallas Ft. Worth area. 

Pilots of autonomous aircraft met controller 
assigned meter fix constraints with high success. 
Separation violations by subject pilots did not appear 
to vary with traffic level and were mainly attributable 
to software errors and procedural lapses. Controller 
workload was lower for mixed flight conditions, even 
at higher traffic levels. Pilot workload was deemed 
acceptable under all conditions. Controllers raised 
several safety concerns, most of which pertained to 
the occurrence of near-term conflicts between 
autonomous and managed aircraft. These issues are 
being addressed through better compatibility between 
air and ground systems and refinements to air and 
ground procedures. 


Introduction 

Background 

Despite recent economic and security concerns, 
demand for air travel is already meeting or exceeding 
previous levels and is projected to increase further. In 
response to these demands, the National Aeronautics 
and Space Administration (NASA) is investigating a 
new concept of operations known as Distributed Air 
Ground Traffic Management (DAG-TM) [1]. 

One component of DAG-TM, En Route Free 
Maneuvering, represents a paradigm shift between a 
centralized ground-based system to a distributed 
system. DAG-TM has the goal of substantially 
improving capacity while maintaining or improving 
safety. Under this concept, flight crews of 
appropriately equipped “autonomous” aircraft fly 
under Autonomous Flight Rules (AFR). These 
aircraft are able to choose their own route and 
altitude, subject to maintaining separation from all 
other aircraft. Controllers continue to provide 
separation between “managed” aircraft unequipped 
for autonomous flight and traffic flow management 
services for all aircraft. En Route Free Maneuvering 
shares a common element with several other long- 
term operational concepts that call for delegation of 
some level of autonomy to flight crews, including 
those proposed by RTCA, Eurocontrol, and Boeing 
[2-4]. 

Over the past several years, NASA researchers 
have conducted several studies to investigate the 
feasibility and benefits of distributed separation 
responsibilities, focusing on both air and ground 
requirements [5-7]. As a continuation of this work, 
NASA Ames and Langley Research Centers 
conducted a joint human-in-the-loop experiment of 
integrated air-ground operations during the summer 
of 2004. This experiment investigated two key 


1 



features of the DAG-TM concept: mixed operations 
and scalability. 

Experiment Motivation and Objectives 

Mixed Operations 

Under DAG-TM, AFR and Instalment Flight 
Rules (IFR) aircraft fly in the same airspace. 
Conducting mixed operations allows equitable 
airspace access to all users while providing benefits 
to those equipping for AFR. Concept feasibility 
requires that air and ground systems and procedures 
be able to accommodate this equipage mix. 

Scalability 

A primary anticipated benefit of DAG-TM En 
Route Free Maneuvering is its ability to substantially 
increase capacity without adversely affecting safety. 
Capacity is increased by adding autonomous aircraft 
while maintaining present levels of managed aircraft. 
In this way, each additional autonomous aircraft adds 
air traffic management capability to the system, 
thereby scaling capacity to meet demand. 

Objectives 

The experiment addressed the following 
research objectives: 

• Investigate the safety and efficiency of mixed 
operations in high-traffic density sectors 
compared to operations with all managed 
aircraft. 

• Investigate the ability to safely increase the 
number of total aircraft in a sector (beyond 
controller manageable levels) if the number 
of managed aircraft remains at or below 
current-day high-density levels. 

Method 

Simulation Facilities 

The DAG-TM simulation environment was 
distributed across two NASA facilities and several 
laboratories as follows: 

Langley Research Center: 

• The Air Traffic Operations Laboratory 
(ATOL) housed 12 subject pilot workstations 
(including the Autonomous Operations 
Planner [8]) as well as the Traffic Manager 
system for handling multiple pseudo-piloted 
aircraft [9]. 

Ames Research Center: 

• The Airspace Operations Laboratory (AOL) 
provided aircraft target generation and the 
Multi-Aircraft Control System (MACS), 


which supported controller workstations and 
multi-aircraft (pseudo-pilot) flight deck 
stations [10]. 

• The Flight Deck Display Research 
Laboratory (FDDRL) provided eight subject 
pilot workstations incorporating the three- 
dimensional Cockpit Display of Traffic 
Information (3D CDTI) [10-12], 

• The Crew Vehicle Systems Research Facility 
housed the Advanced Concepts Flight 
Simulator (ACFS), a high fidelity full 
mission flight simulator that also provided 
the 3D CDTI on both map displays [10]. 

The Aeronautical Datalink and Radar Simulator 
(ADRS) served as the central hub for information 
exchange between pilot and controller facilities at 
Ames [10]. Data exchange between Ames and 
Langley occurred through the ADRS to Langley 
Gateway over a dedicated internet link. At Langley, 
information from the Gateway was distributed across 
a High Level Architecture and made available to all 
other components in the simulation. 

Participants 

Subject participants consisted of 5 professional 
controllers and 22 commercial airline pilots. Each of 
the four en route radar controllers was assigned to a 
single high or low altitude sector, with all positions 
located in the AOL (see airspace description in the 
next section). The fifth controller filled the role of a 
tracker, supporting the radar controllers during peak 
workload periods. All controllers had participated in 
several previous DAG-TM experiments conducted in 
the Ames AOL. Retired pseudo controllers 
monitored traffic within and assisted with handoffs to 
and from the adjoining “ghost” sectors. 

All subject pilots at Ames and Langley were air 
transport rated and had glass cockpit experience. 
Twelve pilots were located in the Langley ATOL and 
operated the subject pilot workstations. They had 
never participated in a previous DAG-TM study. 
Eight other pilots were located in the Ames FDDRL 
and operated the single-aircraft MACS flight deck 
stations. Two additional pilots at Ames flew the 
ACFS as a captain and first officer pair. All Ames 
pilots had previously participated in DAG-TM 
experiments. 

Non-airline pseudo-pilots at both centers 
monitored and interacted with the background aircraft 
in the simulation (those aircraft not flown by subject 
pilots). Pseudo pilots flying AFR aircraft used 
decision support tools to resolve traffic conflicts and 


2 



comply with traffic flow management constraints 
assigned by controllers. When flying IFR aircraft, 
pseudo-pilots implemented all controller instructions. 

DAG-TM Airspace 

The DAG-TM airspace (Figure 1) is a modified 
portion of the airspace in and around Dallas/Fort 
Worth (DFW) Air Route Traffic Control Center and 
DFW Terminal Radar Control (TRACON) facility. It 
was chosen and modified so that subject controllers 
could work continuous traffic streams from en route 
to final approach. The center’s test area included 
three high altitude (Amarillo, Wichita Falls, and 
Ardmore) sectors and one low altitude (Bowie) 
sector. The merge point and TRACON boundary fix 
(BAMBE intersection) were within Bowie airspace. 



Figure 1. DAG-TM Airspace 


Equipage and Flight Operations 

Airborne Capabilities 

Although Ames and Langley incorporated 
different aircraft simulations and flight deck decision 
support tools, all subject and pseudo-piloted aircraft 
at both centers had minimum capabilities consistent 
with their AFR or IFR status. All aircraft used 
Automatic Dependent Surveillance Broadcast (ADS- 
B) to exchange air-air and air-ground state and intent 
information. The centers employed two different 
ADS-B models. Langley aircraft broadcast target 
state information and up to four trajectory change 
points. Ames aircraft broadcast the entire flight plan. 
The Langley model incorporated several data link 
performance limits [13]. Controllers uploaded 
Required Times of Arrival (RTA) at the meter fix to 
AFR aircraft through Controller-Pilot Data Link 
Communications. AFR aircraft had airborne conflict 
management capabilities enabling self-separation and 
free maneuvering as well as the ability to meet 
assigned meter fix constraints. 


Ground Capabilities 

The meter fix arrival scheduler served a dual 
purpose. It prepared the traffic flow into the 
TRACON and provided slots for AFR aircraft to 
merge with IFR aircraft in an orderly manner. The 
scheduler automatically uplinked an RTA to each 
arriving AFR aircraft after it crossed the “freeze 
horizon” 160 NM from the meter fix. This RTA or a 
corresponding Scheduled Time of Arrival (STA) for 
IFR aircraft was held constant after this point. AFR 
aircraft were responsible for making necessary flight 
path changes to meet the RTA, whereas controllers 
issued speed and vector commands to IFR aircraft in 
order to meet the STA. 

The ground stations incorporated conflict 
detection, using a 15-minute time horizon for IFR- 
IFR conflicts. As described previously, AFR aircraft 
were required to resolve all conflicts with IFR 
aircraft. Nonetheless, controllers were notified of 
IFR-AFR conflicts within 3 minutes of predicted 
separation loss as a safety back up. They were not 
required to act on this information nor were they 
alerted to AFR-AFR conflicts. 

A trial planner allowed the controllers to 
detennine the schedule impact and conflict status of 
proposed flight path changes for IFR aircraft. 

Flight Operations 

AFR pilots were required to maintain separation 
from all other aircraft and meet flow management 
constraints assigned by the controller. As in today’s 
operations, controllers separated IFR aircraft from 
each other and provided traffic flow management 
services to all aircraft. Ames incorporated priority 
flight rules for AFR pilots [14]. 

In order to help prevent conflicts and minimize 
their impact, AFR aircraft had to comply with three 
maneuvering restrictions: 1) not maneuver in a way 
that would create a near-term conflict with another 
aircraft (within 4 minutes), 2) maneuver when in 
conflict with a managed aircraft, and 3) maneuver at 
least two minutes before a predicted separation loss. 
Similarly, controllers were not allowed to maneuver a 
managed aircraft such as to create a near- term 
conflict with an autonomous aircraft. 

Experimental Design 

The experiment investigated mixed operations 
and scalability issues by varying AFR/IFR traffic 
ratio and the total sector count. A “within-subjects” 
design was implemented for both pilots and 
controllers. 
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Figure 2 depicts the en route traffic load and 
equipage mix for each of the four experimental 
conditions (C1-C4). “TO” represents a traffic 
threshold that approximates the current-day monitor 
alert parameter for the simulated airspace. “Tl” is a 
projected threshold above which managed-only 
operations cannot be achieved. TO and Tl levels 
were determined in a prior controller-in-the-loop 
study at Ames. Traffic density increases were solely 
due to increases in overflight traffic. Arrival flows 
were kept at a constant, near-saturated level 
throughout the experiment. 



Figure 2. Experimental Design 


Scenario Description 

Each subject pilot scenario consisted of either 
an en route segment (level cruise across Amarillo or 
Ardmore sectors) or a descent to the terminal area 
meter fix (either through Amarillo and Wichita Falls 
or Ardmore). All aircraft were initialized with a 
flight plan and AFR pilots were asked to fly it unless 
it became necessary to resolve a conflict. Arrivals 
were given initial instructions to cross the BAMBE 
meter fix at 250 knots and 11,000 ft. AFR aircraft 
were also required to comply with the RTA issued 
when crossing the freeze horizon. 

Subject pilots flew a total of 16 scenarios (two 
en route and two arrivals) per condition. As part of a 
separate investigation, half of the Langley subject 
pilot arrivals were conducted in a sector southeast of 
DFW airport. Therefore, each Langley pilot flew 
only one arrival in the subject controller sectors per 
condition. Pilots alternated between the Amarillo and 
Ardmore sectors. Scenario order and sector 
assignment were counterbalanced. All subject pilots 
flew as IFR aircraft for condition Cl and AFR 
aircraft for conditions C2-C4. 

Due to differences in ADS-B models, airborne 
system requirements, and priority flight rules, subject 
pilots from Ames and Langley flew in different 


sectors and only interacted when arrival flights 
merged at the meter fix. 

Scripted Conflicts for Langley Aircraft 

To have a set of comparable conditions for 
analysis, carefully constructed conflicts were scripted 
in each scenario. This design allowed a focused 
assessment of key mixed operations and scalability 
performance metrics in a more repeatable and 
controlled environment. For each Cl scenario, six 
IFR-IFR conflicts were scripted for the subject 
controller to resolve (one per Langley subject-piloted 
IFR overflight). For each scenario in C2-C4, six 
AFR-IFR conflicts were scripted for subject pilots to 
resolve (one per Langley AFR overflight) in addition 
to one IFR-IFR conflict. The IFR-IFR conflicts in 
C2-C4 were included so that ground-resolved 
conflicts could be compared across traffic conditions 
in a similar manner as airborne-resolved conflicts. 

The conflicts were all constructed using a 
similar geometry, with a co-altitude intruder 
converging at a 60-deg angle. In addition to globally 
scaling overflight traffic over an entire scenario, 
traffic was locally scaled in a structured manner 
around each scripted conflict via proximate but non- 
conflicting “flanking” aircraft. For Cl and C2 
scenarios, one flanking aircraft was added at -1000 ft 
relative altitude. For C3, a second, opposite-direction 
aircraft was added at +1000 ft relative altitude. For 
C4, a third, co-altitude aircraft was added. Figure 3 
shows the scripted conflict geometry. The flanking 
aircraft were IFR in Cl and AFR in C2-C4. 



Figure 3. Scripted Conflict Geometry 


Training 

Pilots and controllers received two full days of 
training prior to running the data collection scenarios. 
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Training culminated with full mission scenarios 
conducted jointly between the two centers. 

Performance Metrics and Data Analysis 

Data collection focused on objective and 
subjective performance metrics related to concept 
feasibility in the areas of mixed operations and 
scalability. Primary objective metrics included the 
occurrence of separation violations (two aircraft 
closer than the minimum required spacing of 5 NM or 
1000 ft) and conformance with assigned meter fix 
constraints (time (RTA or STA), altitude, and speed). 
Separation violations were only counted when they 
occurred in subject controller sectors and when the 
confederate controller did not contribute to the loss of 
separation (LOS). An aircraft was considered to 
“conform” to the meter fix constraints if it was within 
15 seconds, 300 feet, and 10 knots of the assigned 
values. The three constraints were evaluated 
individually for each pilot. Subjective measures 
included pilot and controller workload and safety 
ratings for each run. Chi square analyses were 
performed on separation violation and meter fix 
conformance percentages. 


Results and Discussion 

Results are presented for Langley subject pilots, 
Ames subject pilots, and controllers. The two flight 
deck sections only include a discussion of subject 
pilot data for the indicated NASA Center. Because 
scripted conflicts were only conducted in sectors 
containing Langley subject pilots, ground-side 
scripted conflict results are presented in the Langley 
Flight Deck section. 

Due to the system and experiment design 
differences described previously, results for each 
flight deck section should be evaluated on their own 
merit, without drawing performance comparisons 
between them. 


Langley Flight Deck Results 

Separation Violations 

Separation violations for Langley subject- 
piloted aircraft as a function of flight rules type are 
shown in Figure 4. 

Of the thirteen total separation violations, one 
occurred between IFR-IFR aircraft, ten between 
AFR-IFR aircraft, and two between two AFR aircraft. 
Six, of the thirteen, separation violations occurred 
during C4. Three additional violations that occurred 
due to small simulation time discrepancies across the 
air/ground network were not included in this analysis. 


Due to limited numbers in each category, a two- 
way chi square analysis was limited to the C3 and C4 
conditions and AFR-IFR and AFR-AFR conflict 
types. This test revealed no significant differences 
between these traffic levels and operation types (X 2 
(2) = 2.22, p> .05). 
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Figure 4. Langley Separation Violations by 
Condition 

A detailed analysis was performed for each case 
to determine the likely contributing factors. Eleven 
out of the thirteen cases were attributed to a “system” 
error. The most prominent of these errors was a 
software bug in the short-term, state -based conflict 
detection system that prevented short-term conflicts 
from being shown to the pilot until after the 
separation loss had occurred. This problem was the 
likely cause of six separation violations. Another 
problem included a known design problem where 
proposed tactical resolutions could exist within a 
displayed “no-fly” region, leaving the pilot with 
ambiguous guidance. A trajectory modeling 
discrepancy and Flight Management System (FMS) 
guidance problem rounded out the system issues. 

Procedural or training problems were deemed to 
contribute to two violations. In both of these cases, a 
pilot maneuvered into a no-fly region. These 
maneuvers caused unrecoverable near-term conflicts 
and were in violation of AFR. Only one of the 
thirteen events was partially attributable to an AFR 
“situation,” defined to be a case where the system 
performed properly for the aircraft in question and 
the pilot followed proper procedures. Even on this 
case, a series of prior system errors helped create a 
difficult situation. 

The lower occurrence of AFR-AFR separation 
violations (compared to AFR-IFR) suggests that 
pilots of other AFR aircraft used decision support 
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tools to recognize the impending conflict and take 
corrective action. Conversely, controllers were not 
asked to monitor AFR-IFR conflicts and they were 
de-emphasized on the ground displays. 

Conformance 

As described previously, the controller assigned 
a crossing time, altitude, and speed at the meter fix 
for each arriving aircraft. (En route overflight aircraft 
were not subject to constraints.) Figure 5 shows the 
percentage of Langley subject-piloted aircraft (across 
all arrival runs) that conformed with the time, 
altitude, and speed restrictions, respectively. These 
values only include arrival flights in subject 
controller sectors. For condition Cl, the time 
reference was the ST A when the aircraft crossed the 
freeze horizon. 


□ Time ■ Altitude □ Speed 



Cl C2 C3 C4 


Condition 

Figure 5. Langley Subject Pilot Meter Fix 
Conformance 

Pilots were predominantly able to meet their 
assigned constraints and there does not appear to be a 
performance degradation as traffic level increases. 
Two of four RTA deviations for AFR runs were 
caused by the pilot entering the wrong RTA into the 
FMS. In one of these cases, the improper time entry 
led to a number of short term conflicts near the meter 
fix, and this pilot also deviated from the assigned 
altitude and speed restrictions. 

Although there appears to be a slight trend 
toward better time conformance for AFR aircraft, this 
disparity may have been due to differences in 
controller procedures needed to accommodate 
different air/ground data link formats associated with 
Langley subject-piloted aircraft flying under IFR. 
These procedures were only needed under condition 
Cl (the only time these aircraft flew IFR.) 
Nonetheless, results from a chi square test indicated 
that no significant differences existed between the 
time conformance levels between Cl and C2 (X 2 (1) 


= 1 . 1 5, /? > .05), nor between C2, C3, and C4, (X 2 (2) 
= 2.18,/) > .05). 

Chi square tests performed on altitude and speed 
revealed no significant differences for mixed 
operations (Cl vs. C2) or scalability (C2 - C4). 

After each AFR scenario (conditions C2, C3, 
and C4), pilots completed an arrival or en route 
questionnaire, as appropriate. 

Scripted Conflicts 

Sixty-nine of 72 AFR-IFR scripted conflicts 
occurred. While each aircraft trajectory was designed 
to be conflict-free until the scripted conflict, 
unpredictable traffic interactions prevented some 
conflicts from taking place. Objective data were used 
to assess the effects of traffic density on the following 
airborne performance measures: time before 

predicted LOS at which conflicts were resolved, the 
frequency of induced conflicts (defined as additional 
non-scripted conflicts caused by a resolution 
maneuver), and the frequency of conflicts requiring 
multiple maneuvers for resolution. 

Mean times to predicted LOS when conflicts 
were resolved for subject pilot scripted AFR-IFR 
conflicts changed little as a function of traffic 
condition. The values were Mean (M) = 486 s, 
Standard Deviation (SD) = 87 s (C2); M = 473 s, SD 
= 97 s (C3); and M = 446 s, SD = 146 s (C4). A 
within-subjects ANOVA indicated that the slight 
decrease was not significant (F( 2,22) = 0.82, p > 
0.05). AOP provided a ten-minute conflict look- 
ahead time; under all traffic conditions tested, pilots 
were generally able to resolve conflicts before 5 
minutes to LOS (at which time the AOP alerting level 
increased) and well before 2 minutes to LOS 
(minimum required time to commence maneuver). 

Seven induced conflicts resulted from resolution 
maneuvers for AFR-IFR scripted conflicts. All of 
these occurred in C4. It is a concern that induced 
conflicts increased in the highest-density traffic 
condition, possibly indicating a reduction in system 
stability. However, it should be noted that all but one 
of these induced conflicts was caused by a manual 
maneuver, for which the pilot did not seek or use 
AOP resolution guidance. Thus, induced conflicts 
may be mitigated through more extensive pilot 
training and modifications to AFR procedures. 
Human factors issues have also been identified with 
the AOP conflict prevention tools, designed to alert 
pilots to potential conflicts along proposed manual 
maneuvers. For example, in the AOP version used in 
this experiment, computer limitations required the 
conflict prevention symbology to “time out” on the 
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aircraft displays, which occasionally masked the 
existence of potential conflicts. 

Figure 6 shows, in each traffic condition, the 
percentage of AFR-IFR scripted conflicts that 
required multiple resolution maneuvers. While the 
rate of multiple resolutions increased slightly as 
traffic increased, a x 2 test showed these differences 
were not significant (x 2 (2, N= 69) = 0.41,/) > 0.05). 
Improvements to AOP logic, such as the addition of 
spatial or temporal buffers for resolution maneuvers, 
may reduce the incidence of conflicts requiring 
multiple resolutions, especially near trajectory 
inflection points (such as top-of-descent) where 
aircraft dynamics are known to vary unpredictably. 
No such buffers were used for this experiment. 
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Pilot Questionnaires 

Across all conditions, pilots overwhelmingly 
found that AFR flight operations did not adversely 
affect their workload. On a seven point scale ranging 
from 1 (completely acceptable) to 7 (completely 
unacceptable), average workload levels for 
overflights were 1.5, 1.5, and 2.3 for C2, C3, and C4 
respectively. They were only slightly higher for 
arrival flights with means equal to 2.5, 2.2, and 2.4 
for C2, C3, and C4 respectively. 



270-330 330-390 390-450 450-510 510-570 570-630 


Time to LOS at Resolution (sec) 


Figure 7. Time to LOS at Conflict Resolution 
(Controller-Resolved Scripted Conflicts) 


Traffic Condition 

Figure 6. Conflicts Requiring Multiple Resolution 
Maneuvers (Pilot-Resolved Scripted Conflicts) 

Overall, the AFR-IFR scripted conflict results 
support the hypothesis that airborne conflict 
resolution performance does not significantly degrade 
under high-density traffic conditions. 

Thirty-one of 36 IFR-IFR scripted conflicts 
occurred. Due to the small number of data points in 
the mixed-operations scenarios (3 in C2, 3 in C3, and 
2 in C4 vs. 23 at Cl), statistical analyses were not 
performed on these data. However, using the same 
metrics previously presented for airborne-resolved 
scripted conflicts, the effects of mixed operations and 
traffic density on ground-side resolution performance 
can be qualitatively assessed. 

Figure 7 shows a histogram of times to 
predicted LOS at conflict resolution for controller- 
resolved, IFR-IFR scripted conflicts in C2-C4. No 
consistent trend is observed across traffic conditions 
and resolution times are comparable to times in Cl 
(M = 490 s, SD = 111 s). Also, none of the IFR-IFR 
scripted conflicts required multiple resolution 
maneuvers, and no resolution maneuvers caused an 
induced conflict. Qualitatively, these results indicate 
that mixed operations (in C2) and increased AFR 
traffic (in C3-C4) had little effect on the ability of 
controllers to resolve IFR-IFR conflicts. 


Pilots also reported a high level of safety across 
all three AFR conditions. Traffic level and type of 
scenario (en route or arrival) had almost no effect on 
reported responses to this question. On a seven point 
scale ranging from 1 (completely unsafe) to 7 
(completely safe), means were [C2, C3, C4] = [6.1, 
6.4, 6.0] for overflights and [C2, C3, C4] = [6.1 6.3, 
6.1] for arrivals. During the debriefing, most pilots 
expressed high confidence in AFR operations during 
en route flight, but some had reservations about 
implementing AFR during descent. Reliability of the 
AOP (for the version used during the experiment) and 
traffic congestion near the meter fix were cited as 
concerns. 

Ames Flight Deck Results 

Separation Violations 

There were no LOS incidents for the Ames 
subject-piloted AFR aircraft. There were 139 
conflicts resolved. Of these, 122 were detected prior 
to four minutes to conflict. Figure 8 shows 17 late 
alerts (less than four minutes to LOS). Consistent 
with the increased traffic load, the majority (11 late 
conflicts) occurred in the C4 condition. All 17 were 
due to an aircraft executing a maneuver which 
brought about the late alert. However, a maneuver by 
an Ames subject-piloted AFR aircraft was 
responsible for these late conflicts (concept 
violations) in only four instances. Furthermore, in 
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three of the four cases, a flaw occurred in the conflict 
resolution software (indicating to the pilot that the 
maneuver was conflict free, when it was not). The 
reason for the remaining apparent procedural error 
remains to be determined. 



C2 C3 C4 

Condition 


Figure 8. Number of Conflicts Detected vs. Time 
to LOS and Condition 

Figure 9 shows a total of two conflicts with late 
resolutions (under two minutes to LOS), with one of 
them due to a software flaw in which the conflict 
detection did not function properly. The other 
instance was associated with the aforementioned 
incident where the pilot maneuvered into a conflict at 
some point between two and four minutes to LOS, 
with the result being that the pilot was not able to 
resolve the conflict until there was less than two 
minutes until the projected LOS. 

■ Under 2 Minutes 



Condition 

Figure 9. Number of Conflicts Resolved vs. Time 
to LOS and Condition 


Figure 10 shows the amount of time needed 
to resolve burdened conflicts. While not an integral 
part of the concept, the Ames AFR pilots were asked 


to resolve all burdened conflicts within two minutes 
(120 sec) of receiving a conflict alert. Figure 10 
shows that this criterion was met approximately 89% 
of the time. 



Time in Alert 

Figure 10. Distribution of Times Required to 
Resolve Burdened Conflict Alerts 

Finally, Figure 11 shows the number of times 
the ownship and intruder resolved the conflict as a 
function of burdening [14]. While the IFR aircraft 
almost never resolved a conflict (it was never 
burdened), it is noteworthy that almost 1/3 of the 
non-burdened aircraft resolved the conflicts in AFR- 
AFR conflicts. 



AFR (Burdened) AFR (Not IFR 

Burdened) 


Intruder 

Figure 11. Number of Conflict Resolutions for 
Ownship and Intruder as a Function of Burdening 


Conformance 

Figure 12 shows Ames subject-pilot AFR meter 
fix conformance. There was no significant difference 
in objectively measured performance as a function of 
any of the conditions. Therefore, it does not appear 
that mixed operations, nor increased en route traffic 




load had any effect on the ability of the arriving 
flights to meet the meter fix constraints. Among the 
AFR flights, one failed to meet the speed constraint, 
and one failed to meet the altitude constraint, but all 
met the RTA constraint. 


□ Time m Altitude □ Speed 



Condition 


Figure 12. Ames Subject Pilot Meter Fix 
Conformance 

Subjective Assessments 

The subjective assessment of pilot workload 
was measured following each run using the Modified 
Cooper Harper (MCH) workload scale. The MCH 
allows for ratings between 1 (Very easy/workload 
insignificant) and 10 (Impossible/task abandoned, 
unable to apply sufficient effort). Pilot responses 
across all simulation trials ranged from 1 to 6. 
However, approximately 98 percent of responses 
ranged from 1 to 3. In order to receive a rating 
between 1 and 3, it must be possible to complete the 
task and workload must be perceived as tolerable and 
satisfactory. Ratings from 4 to 6 suggest that task 
workload is high but not high enough to impact 
performance on the primary task. 

Figure 13 shows the average workload ratings 
of arrivals and overflights in each condition (+1 SD). 
Not surprisingly, workload ratings were higher for 
the arrival flights than for overflights in all 
conditions. Pilots also responded unanimously with 
the lowest possible workload rating (1) for the 
managed overflight runs (Cl). In addition. Figure 13 
shows an increase in perceived workload from all 
managed (Cl) to mixed (C2-C4) operations for both 
flight types, but workload remained acceptably low. 

At the completion of the simulation, pilots were 
asked to make preference comparisons between the 
AFR conditions (C2-C4) and the IFR condition (Cl) 
on four different dimensions: overall safety, overall 
workload, ease of time conformance, and overall 
situation awareness. 


□ Arrival ■ Overflight 


Figure 13. MCH Workload Ratings as a Function 
of Flight Phase and Condition 

Pilot preferences for each question were 
analyzed using the Analytic Hierarchy Process (AHP) 
statistical technique [15], wherein the preference data 
for each question is transformed into a percentage 
and averaged for all pilots to produce numerical 
ranking scores. Pilots on average preferred the AFR 
conditions in terms of overall safety, ease of meeting 
the RTA/STA, and overall SA. Pilots preferred the 
IFR condition in terms of overall workload. The 
average ranking scores are presented in Figure 14. 
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Figure 14. Ames Subject Pilot Rankings 

Ground-Side and Controller Results 

This section provides overall data for aircraft 
controlled by subject pilots (Ames and Langley) and 
those managed by subject controllers. Pseudo aircraft 
data are not included. 

Separation Violations 

Overall separation violations are shown in 
Figure 15. Although the number of violations 
increased gradually from C1-C4, it is difficult to 
generalize the results from this number because each 
violation resulted from a unique circumstance. 
Previous sections on the air side results summarized 
the circumstances that led to IFR-AFR and AFR- 
AFR violations. Six of the seven IFR-IFR separation 
violations were linked to a pseudo pilot error in 
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executing its clearance or a pseudo controller 
breaking procedures by handing off an aircraft in 
conflict to a subject controller. The absence of 
subject controller-caused separation violations 
suggests that the increasing traffic levels of AFR 
aircraft did not negatively impact their ability to 
separate IFR aircraft. 
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Figure 15. Overall Separation Violations 
Conformance 

Figure 16 provides the total number of time, 
altitude, and speed non-conformance events for 
arriving aircraft. The small difference in time 
deviations in Cl and C2 (6 and 2, respectively) 
suggest that mixed traffic had little effect on time 
conformance. Likewise, time deviation differences in 
C2, C3 and C4 (2, 4, 1, respectively) show no trend 
in the number of deviations as overflight traffic 
increased. Chi square tests revealed that none of 
these differences were significant. 
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Figure 16. Overall Meter Fix Conformance 

IFR aircraft were equally likely to conform to 
the crossing restrictions regardless of whether or not 
autonomous aircraft were in the arrival flow. These 
results suggest that mixed operations did not hinder 
traffic flow management of arrival flights. 


The additional AFR traffic in C3 and C4 were 
all overflights, therefore the impact of scalability on 
arrival conformance was expected to be minimal if 
interactions between arrivals and AFR overflights 
were negligible. The results revealed that arrival 
conformance was not significantly affected by the 
scaled overflight traffic. Time, speed, and altitude 
restrictions were met with regularity across 
conditions, suggesting that controllers and AFR pilots 
were able to handle any extra maneuvering caused by 
AFR overflights. 

Controller Workload Assessment 

Controller workload is one of the key metrics to 
examine the potential benefits of En Route Free 
Maneuvering. It is presumed that controller 
workload is a key limiting factor to the number of 
aircraft that can safely be managed for a given sector. 
One hypothesis for this study is that offloading the 
separation responsibilities to AFR aircraft has the 
potential to increase the overall aircraft count beyond 
what controllers could manage if all flights were IFR. 

Subjective workload assessments were collected 
from controllers using the Air Traffic Workload Input 
Technique [16]. Controllers were asked to rate their 
workload using Workload Assessment Keyboards on 
a scale of 1 to 7, at 5-minute intervals throughout 
each simulation run. 

Figure 17 presents the average workload rating 
for each of the four controllers in each experimental 
condition (± 1 standard deviation). The small number 
of data points (4) in each condition prevents us from 
determining if the differences between conditions are 
significant. However, a visual inspection of the 
graph indicates that for each controller, workload 
ratings were the highest in Cl. The higher average 
workload in Cl, as compared to C2, suggests that the 
presence of AFR flights did not negatively impact 
perceived workload. 

The ratings in C2, C3 and C4 reflect the 
influence of increasing traffic through the addition of 
AFR overflights. It was expected that no appreciable 
difference would exist in the Bowie sector because no 
overflights were present in that sector. Workload 
ratings in C2, C3, and C4 for the Amarillo and 
Ardmore controllers showed a slight increase. 
However, all average workload ratings were under 
four, indicating that increased workload was quite 
modest. This observation is particularly interesting 
because the volumes of traffic presented in C3 and 
C4 were considered not to be manageable under all- 
managed operations. For example, total aircraft 
count in C4 was approximately double the count in 
Cl and C2, and yet the workload was lower in C4 
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than Cl. This finding supports the hypothesis that 
controllers can handle much larger volumes of traffic 
than in current day operations if additional traffic 
consists mostly of AFR flights. However, the lack of 
workload differences does not suggest that the 
increase in overflights did not adversely impact 
controllers, as there were a number of safety issues 
and concerns, which will be described in the next 
section. 



Controller 


Figure 17. Average Controller Workload Ratings 

Controller Safety Assessment 

The controller participants were asked to make 
pairwise preference comparisons between all possible 
pairs of simulation conditions with respect to overall 
safety. These comparisons were analyzed using AHP 
to determine a ranking for each condition. Figure 18 
summarizes the safety rank scores of condition by 
each controller position. The data show that ranking 
the four conditions from most to least safe, the 
controllers consistently ranked the all managed 
condition (Cl) as the safest and the L3 -mixed 
condition (C4) as the least safe. 
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Figure 18. Average Controller Safety Assessment 

The most common concern raised pertained to 
AFR-IFR conflicts. Although conflicts and 
separation violations between AFR and IFR aircraft 
changed little across traffic levels for the subject 
pilots, these events increased dramatically for pseudo 


pilots under condition C4. As discussed previously, 
controllers were notified of AFR-IFR near-term 
conflicts within 3 minutes of predicted separation 
loss. 


In general, controllers felt that AFR-IFR 
conflicts posed a safety problem because 1) AFR 
aircraft did not always resolve conflicts in a timely 
manner, 2) it was not always apparent that the AFR 
aircraft was taking action, and 3) it was not always 
clear what the AFR aircraft would do. 

Although controllers had been briefed that 
resolution of AFR-IFR conflicts was not their 
responsibility at any time, they often felt compelled 
to act when presented with a short-term conflict that 
had not been resolved. This mindset left controllers 
short-handed, due to the lack of available maneuver 
intentions from the AFR aircraft and the need to 
suddenly become involved in a situation to which 
they had previously paid little attention. 

These issues highlight the importance of clear 
and unambiguous procedures for both pilots and 
controllers when handling short-term AFR-IFR 
conflicts. If resolution responsibility is to be shared 
between the pilot and controller under these 
situations, then some level of air and ground system 
compatibility is required. Alternatively, if 
responsibility is to remain solely with the AFR pilot, 
then the decision to alert the controller to these 
conflicts should be re-visited. 

Additional issues raised by controllers include 
the need for procedures that handle failures of AFR 
conflict management capabilities. Controllers also 
had some concern about their ability to maneuver IFR 
aircraft in the presence of additional AFR aircraft. 
Scripted conflict analysis suggests that AFR aircraft 
did not affect a controller’s ability to separate IFR 
traffic. 


Conclusions 

The DAG-TM concept element of free 
maneuvering has great potential to increase en route 
and transition airspace capacity, provided that safety 
concerns raised by controllers can be addressed. This 
large scale, high fidelity experiment with pilots and 
controllers has indicated that many autonomous 
aircraft can be added to a moderately high number of 
managed aircraft in the same airspace without 
increasing controller workload, while maintaining 
flight crew workload at acceptable levels. 
Comparable controller workload benefits associated 
with some level of aircraft self-separation have also 
been reported by NLR and Eurocontrol [17-18]. In 
this study, controller workload appeared to correlate 
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primarily to the number of managed aircraft, whereas 
the number of autonomous aircraft in the airspace had 
far less impact. Mixed operations provided 
equivalently efficient arrival flows as managed 
operations, suggesting that time -based traffic 
management constraints are an effective way to 
coordinate managed and autonomous flights from 
mixed into managed airspace. 

The experiment has also demonstrated that a 
well-integrated and compatible set of advanced 
air/ground automation and well-defined procedures 
will be required to enable the capacity gains without 
compromising safety. Separation losses, close calls, 
and late and false alerts encountered, especially 
between managed and autonomous aircraft, prompted 
controllers to rate mixed operations much less safe 
than managed operations. In most of the safety 
compromising instances, controllers were aware of an 
upcoming problem but unable to determine the flight 
crew’s intention based on the data linked information 
or via a voice inquiry. Inadequate air/ground system 
and procedural compatibility concerning short-term 
managed/autonomous conflicts resulted in 
uncoordinated maneuvers and potential safety 
hazards. Pseudo pilot ability to effectively maneuver 
increased numbers of pseudo aircraft at higher traffic 
levels may also have played a role. Controllers’ 
safety concerns increased with the number of 
autonomous aircraft in the airspace. 

Many pilots were able to maintain safe 
separation at all times and meet their traffic flow 
constraints precisely. In general, pilots did not share 
controllers’ safety concerns regarding autonomous 
operations and reported that they felt comfortable 
managing separation and air traffic constraints with 
appropriate airborne automation and information on 
the other traffic in the airspace. Pilots and controllers 
reported that the all managed condition simulating 
advanced trajectory-based operations with data link 
was safe and would be greatly preferable to current 
day operations. 

Distributing roles and responsibilities between 
pilots and controllers has the potential to greatly 
increase airspace capacity. In order to realize these 
benefits, the following key areas will be considered 
for follow-on research: 

• Harmonization of trajectory specifications 
and processing among all air and ground 
system components. 

• Further investigation of the value of strategic 
conflict resolutions that avoid short-term 
conflicts. 


• Development of compatible conflict detection 
and resolution algorithms between air and 
ground systems when short-term conflicts 
appear between AFR and IFR aircraft. 

• Definition of clear procedures for 
autonomous/managed aircraft interactions 
and for transitions from autonomous to 
managed status and vice versa. 

• Investigation of system failures in the 
operational environment. 

Results of this and follow-on studies, as well as 
efforts to gain operational experience with key 
enabling air and ground technologies will help enable 
progression toward eventual concept implementation. 
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